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(57) Abstract: According to one aspect of the invention, a method 
300 is provided in which a user in a system is allowed to specify a relative 
preference for each representation in a set of representations in which 
information generated from an application domain can be presented 
to the user. The user is allowed to configure one or more parameters 
associated with each representation to quantify one or more render- 
ing features corresponding to the respective representation in one or 
more modalities in which information from the application domain 
can be presented to the user. In response to an information presen- 
tation request to present information to the user, one or more com- 
binations of representations and modalities are selected to be used 
for the presentation request that can be accommodated by available 
resources, based upon a set of criteria including the user preferences 
and resource consumption estimation associated with the presenta- 
tion request. 
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METHOD, APPARATUS, AND SYSTEM FOR DETERMINING 
INFORMATION REPRESENTATIONS AND MODALITIES BASED ON USER 
PREFERENCES AND RESOURCE CONSUMPTION 

FIELD OF THE INVENTION 

The present invention relates to the field of multimodal human-computer 
interaction and interface. More specifically, the present invention relates to a method, 
apparatus, and system for determining information representations and modalities based 
on user preferences and resource consumption requirements. 

BACKGROUND OF THE INVENTION 

The field of human-computer interaction and interface has been undergoing rapid 
changes in recent years due to/advances in computer technology and related fields. In, , 
particular, a lot of time and effort has been directed to utilizing technological advances in 
multimedia and multimodal processing to improve the usability and productivity of 
processing systems. Multimedia systems have been developed and improved at a rapid 
pace to provide the users with more and more options and flexibility in their interface 
with the systems, for both input and output applications. However, current multimedia 
systems still have a number of shortcomings in their approach to using multimedia 
capabilities available to present information in various output modalities. Current 
multimedia systems typically predefine or prescribe the modality and representation for 
presenting any given output information in a fixed or predetermined manner. For 
example, a word processing application may be configured to always present output 
information to the users in text format on the display screen or a company announcement 
system may be designed to always present company announcements as audio output, etc. 
Even when a system or an application is designed to present certain information 
simultaneously in multiple modalities (e.g., both visually and aurally), such a system or 
application typically cannot determine whether such a multi-modal presentation of 
information (e.g., both visually and aurally) in certain situations is desirable or even 
acceptable from a user's point of view. For example, assuming that a company 
announcement system is designed and configured to always present company 
announcements both visually (as text on display screen) and aurally (as audio output on 
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speakers attached to the user's computer), such an inflexible selection of output modality 
(or modalities) may not be desirable or acceptable to certain users in certain situations 
For example, a user A who currently is listening to music or having a conference call ' 
may not want to hear a company announcement as audio output on the speakers In this 
case, user A may just want to see the display of the company announcement on' the 
computer screen. Similarly, a user B who is working on a word document and already 
havmg a number of windows open on his computer screen may not want to see his 
screen to be cluttered with any more text but rather may only want to hear the company 
announcement as audio output. In other words, for any given system and/or application 
ngid or mflexible designation of information to any particular (output) modality may not 
be desrrable and even unacceptable to certain users in certain circumstances or situations 
Too much information presented fa any given output modality may lead to information 
. .. overlpad^ith respect to a user's sensory and mental- capacity to absorb information In 
add-on, certain users may not be able to even receive or interpret information in certain 
modahties. For example, audio output may not be acceptable for users with hearing 
"npan-ment, Likewise, visual output (e.g., text, graphics, video, etc.) is not acceptable 
to users who are blind. In addition to the shortcomings mentioned above, current 
multimodal and multimedia systems also lack, capabilities to effectively and efficiently 
support multiple representations of information in a multimodal environment For 
example, output information from a given application may be represented in different 
forms or formats. However, current systems and apphcations are unable to determine 
wh,ch representations) of information may be better or more preferable from a user's 
pomt of View under certain circumstances. Even when an application is designed to 
prov.de and support alternative representations of information, assignment or selection of 
a particular type of representation is typically predetermined or requires a user to 
manually intervene or specify the particular representation that is suitable for that 
particular user at a particular moment in time. For example, a typical file management 
system may allow the users to view their file listing either as a summa,y listing (e g a 
Ustmg of file nam e S only), a detailed listing (e.g., a listing of file names with additional 
file information such as the file types, the file sizes, etc.), a list of small icons, or a list of 
large .cons, etc. However, a user is typically required to specify or select a particular 
type of listing (representation of file information) that he wants to use at any given 
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moment. Similarly, even when the systems and applications support multiple window 
sizes and positions for visual display, the size (and the position) of the window for any 
given information is either predetermined by the system or application, or the user is 
required to manually select or specify the desirable size (and the position). For example, 
when a user brings up multiple applications, the information generated from the multiple 
applications are either displayed in all large windows'or all small windows in fixed 
positions, or in a fixed overlapping manner without taking into considerations the user's 
preferences or needs and the changes in the system's conditions and environment. 
BRIEF DESCRIPTION OF THE DRAWINGS 

The features and advantages of the present invention will be more fully 
understood by reference to the accompanying drawings, in which: 

t ■ Figure 1 is a.diagram showing a .traditional multimedia/multimodal configuration; 

Figure 2 is a block diagram of one embodiment of a system according to the 
teachings of the present invention; and 

Figure 3 is a flow diagram of one embodiment of a method according to 
teachings of the present invention; and 

DETAILED DESCRIPTION 

In the following detailed description numerous specific details are set forth in 
order to provide a thorough understanding of the present invention. However, it will be 
appreciated by one skilled in the art that the present invention may be understood and 
practiced without these specific details. 

In the discussion below, the teachings of the present invention are utilized to 
implement a method, apparatus, system, and machine-readable medium for managing 
information representation in a multimodal environment. In particular, the teachings of 
the present invention are utilized to implement a decision making process for selecting . 
desirable combinations of information representations and modalities for presenting 
information to the users, based upon their representational. and/or modal preferences. In 
. one embodiment, a user in the system is allowed to specify a relative preference for each 
representation in a set of available representations in which information from a particular 
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application domain can be presented to the user. In one embodiment, the user is also 
allowed to specify a relative preference for each modality in which information from the 
particular application domain can be presented to the user. In one embodiment, the user 
is allowed to configured one or more parameters associated with each representation to 
quantify one or more rendering features of the respective representation with respect to 
one or more modalities in which information from the particular application domain can 
be presented to the user. In response to a new information presentation request to 
present information to the user, one or more combinations of representations and 
modalities are to be selected for the new presentation request that can be accommodated 
by available resources, based upon a set of criteria including the user representational 
and modal preferences and resource consumption estimation for the new presentation 
request. In one embodiment, the selection of the one or more combinations that can be 

accommodated by available resources includes determining whether.ayailable resources 

- • « - -. - »- 

are sufficient for a particular combination of representation and modality, starting with 
the combination that is considered most preferable by the user until available resources 
are sufficient to accommodate at least one combination of representation and modality or 
until all combinations have been exhausted. In one embodiment, a list of rendering 
representatives is maintained which is used to determine the amount of available 
resources. Each rendering representative corresponds to a previously commenced 
presentation request. In one embodiment, the one or more modalities include visual 
modality and aural modality. In one embodiment, the user representational and modal 
preferences are stored in a first database. The parameters associated with each 
representation are also stored in the first database. In one embodiment, the list of . 
rendering representatives is stored in a second database. The teachings of the present 
invention are applicable to any scheme, method and system for information presentation 
and representation in a multimodal environment which includes visual and aural 
modalities. However, the present invention is not limited to visual and aural modalities 
and can be applied to methods, schemes, and systems for information presentation and 
representation in other modalities as well. Likewise, while the discussion herein is 
focused on output modalities of information presentation and representation for 
illustration and explanation purposes, the teachings of the present invention are also 
applicable to input modalities. 
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Figure 1 shows a block diagram of a conventional or traditional 
multimedia/multimodal system 100 that prescribes or predefines the modality and 
representation in which any given output information is presented to the users in a fixed 
or predetermined manner. As shown in Figure 1, each application is designed to convey 
output information in a predetermined manner with respect to the modality and the 
representation in which information can be presented. While an application may be 
capable of presenting or conveying information in multiple representations and/or 
multiple modalities, the modality and representation for conveying any given 
information are typically predetermined and can only change with the user's manual 
intervention or reconfiguration. In addition, the selection of a particular modality and a 
particular representation for presenting or conveying information is made without 
considering individual user's representational and modal preferences and without 
considering changes in the system's conditions and environment. For example, ; ;* 
application A, by default, would always present output information using the 
representation 1 in modality 1 even though representation 2 in modality may be much 
better and more desirable for certain users under certain circumstances. Furthermore, the 
rendering parameters associated with each representation and each modality are also 
fixed or predetermined which may not be preferable or acceptable to certain users under 
certain conditions. For example, assuming that representation 1 is an icon representation, 
it will be displayed, by default, in the same size (predetermined) at the same position 
(predetermined) regardless of how much other information are being displayed on the 
same screen at that point in time. Such a fixed rendering of representation 1 in modality 
1 may be undesirable or unacceptable to a given user under certain circumstances. . 

The following example is provided to illustrate and explain the various aspects of 
the present invention. Consider a simple example of a "Calendar" application a good 
implementation of which will need to support multiple representations or alternative 
displays. In one embodiment, the various representations or displays may include the 
following: 

1 . Bit representation: a minimalist image showing whether or not any appointments 
are scheduled for the remainder of the day. For example, an image of "open 
book" in this Bit representation indicates that one or more appointments still need 
to be attended to. 
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2. Icon representation: a SmaJ , image sho ^ ng just the ^ rf ^ ^ 

3. Bnef representation: a 2-3 line reminder concerning the next schedule 
appointment, giving some level of details. 

4. Daily representation: a lis, of the day's appointments a„ d their times , ^ me 
references a corresponding brief representation 

1 Daily-In-Brief representation: an iconic summary of rite day's appointments and 
their times. 

6. Weekly representation. 
. 7. Monthly representation. 
8. Annual representation. 

In one embodiment, a Hat L may be used to pack all available representations for 
calendar as follows: 

L - < Bit, loon, Briefly, Daily-In-Brief, Weekly, Monthly, Annual* . . 
Whtoh representations) for a calendar should a system display or present to the 
users, gtven mujripie representation in which information associated with the calendar 
appheatton can be presented to the users' The answer to mis question may depend upon 
the types of users and the giving rime as well. For example, a busy executive migh, 
prefer the daily version, while his secretary migh, wan, the weekly or monthly i„ s ,ead 
An e„ g ,„eer migh, wan, to clear moa, of (he display screen for a detailed visualization 
and ,o allocate jua, minimal space fot the icon representation, e,o. 

These varied requirements or preferences by different users may be 
accommodated by expanding the List L shown above to include a relative "priority 
mdex" or "preference index" (also called preference score, for each repression The 
pnonty mdex or preference score for each representation may be preset by the user in a 
user preference database, which is described in more detail below. Assuming tha, the 
exenuttve has specified a priority index or preference score for each representation of the 
calendar, the expanded and improved lis, L for the executive may look like rite following: 

L = < (Bit, 10), (Icon, 10), (Brief, 1), (Daily, 0), (Daily-In-Brief, 3), (Weekly, 2), 

(Monthly, 4), (Annual, 10) > 
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In the present example, it is assumed that a lower priority index or preference 
index corresponds to a higher degree of preference (e.g., a representation with a priority 
index of "1" is considered more preferable or more desirable than a representation with a 
priority index of "3", etc.). Of course, the priority index or preference index can be 
defined so that a higher priority index corresponds to a higher degree of preference. In 
the present example, the preferred representations have lower index values. Accordingly, 
the assumption in above example is that the executive would be most interested in seeing 
the daily representation whose corresponding priority index is 0 and would be much less 
interested in seeing the "Bit", "Icon", and "Annual" representations whose 
corresponding priority index is 10. 

The list L may then be reordered or sorted in a specified order. In the present 
example, the list L is sorted by priority index values in ascending order. The sorted list L 
in ascending order of the priority index values will be as follows: • 

L = < (Daily, 0), (Brief, 1), (Weekly, 2), (Daily-In-Brief, 3), (Monthly, 4), (Bit, 10), 

(Icon, 10), (Annual, 10) > 

The list L of course may be sorted in a different order, for example in a 
descending order of priority index values. 

It is assumed that the above representations can all be presented or displayed in 
the visual modality. While all of the alternative representations for a calendar described 
above lie in the visual modality, they do not all consume the same display or presentation 
resources. For example, the amount of screen real estate (e.g., window size) required for 
each representation may be different. When the screen is mostly empty it probably will 
not matter much which representation(s) are chosen for the calendar. However, when 
numerous windows are open on the screen, then one of the alternatives having smaller 
size will normally be more preferable. To effectively reduce display clutter and improve 
the overall usability of any particular modality and representation, the system may need a 
built-in mechanism for tracking and quantifying the amount of screen real estate 
currently consumed (which is one of the display or presentation resources required for 
rendering). To support this functionality, the list L given above is expanded again, so 
that its entries are now triples whose third component measures the relative sizes of the 
various representations as follows: 
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L - < (Daily, 0, Area=Large), (Brief, 1, Area=Medium), (Weekly, 2, Area=Large), 
(Daily-In-Brief, 3, Area=Medium), (Monthly, 4, Area=Large), (Bit, 10, Area=Small), 
(Icon, 10, Area=Small), (Annual, 10, Area=Large) > 

For a given family of representations, the concept and idea described above can 
be extended to include other important measures of display or presentation resources 
consumed for each representation. The result is that the entries in the list L are no longer 
triples but rather quadruples as shown below: 

L = < Representation name, priority index, modality 1, modality 2> 

- whe ^ t ^ ,a ? t two ^PJ^BS 1 ^^ ^^^^^^ lists whose 
entries are of the form: 

Resource=amount consumed. 

In the above example, the modality 1 notation refers to a list of various types of 
resources consumed or required for rendering the respective representation in modality 1. 
The modality 2 notation is used to refer to a list of various types of resources consumed 
or required for rendering the respective representation in modality 2. In the present 
example, the modality 1 corresponds to visual modality and the modality 2 corresponds 
to aural modality. 

The type of resources in the corresponding modalities is a cognitively relevant 
feature for rendering representation such as: area, position, duration, color, frequency, 
etc. 

To illustrate the usage of visual and aural modalities, suppose now that the 
calendar application wants to alert the user as the next appointment draws near. This can 
be achieved by various means as follows: 

• Use of animation on screen using one of above 8 representations. 

• Use of sound: either through an earcon or voice output. The voice can announce a 
detail appointment while the earcon can herald an impending meeting. 
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Based upon the description and explanation provided above, a high-quality 
multimodal and multi-representational calendar application will be expected to make the 
following decisions on its own: 

o Which representations should be used in accordance with the current display 

context. 

o Where they should be located on the screen. * 

o What type of modalities should be employed? Visual? Voice? Earcon? Or 
combination? 

The following discussion is focused on expanding the ideas and concepts 
described above with respect to the calendar application example to more generalized 
cases in which preferable or desirable combination(s) of representations and modalities 
can be selected for presenting information to the users based upon their preferences of 
Representation's and/qrmodalities' and. whether, ayaikble resources are sufficient to** 
accommodate such combinations. 

Well-established theory shows that for a given application domain (e.g., the 
calendar application, etc.) about half a dozen parameters suffice to quantify the 
cognitively relevant features for rendering a corresponding representation in each output 
modality. The parameters which are used to quantify the rendering of the corresponding 
representation for the visual and aural output modalities may be denoted as follpws: 

{V i} i < Nv ; {a i} i < Na (Nv, Na » 6) 
where V and a are used to denote the visual and aural modalities, respectively. For 
example, using one of the calendar representations described above, some of the relevant 
parameters in the visual modality may include the area or size of the display, the position 
of the display, etc. According to the teachings of the present invention, the users are 
allowed to configure or modify the parameters based upon their preferences to further 
refine the relevant rendering features associated with each representation. The user's 
preferences with respect to the various representations and associated parameters may be 
stored in a database (also referred to as the user preference database herein). For 
example, the user preference database may be used to store the user specified priority 
index (preference index) for each representation and also the user specified parameters 
" associated with each representation. 
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Thus, the family of all possible presentations or displays for a given application 
domain exists within the 6-spaces whose coordinate axes are the V / and a / . 

In one embodiment, each representation in a given application domain can be 
described as a frame or semantic frame in a semantic network (also called the display or 
presentation semantic network herein). Accordingly, the display semantic network can 
be degenerated to a linear list of frames, each of which contains a finite sequence of 
quadruples slots. For a given semantic frame S having a representation K, the 

degenerated linear list L k s may be denoted as follows: 

^=<^(/ a ),p s k 3 f;^;,>(k>o) 

where: 

• Eg is the application-defined handler, which is executed to generate the required 

representative (s), and t A is the relative, elapsed time .from the^monient when the 1 
application issued this presentation or display request. 

• P s k is a priority index, which either comes from the application preset or from 

the user preferences database. As described above, the linear list L k £ may be 

sorted in an increasing or ascending order in accordance with . Hence, E* s 
is the preferred rendering representative among all possible alternatives or 
representations in If s 

• V* and A* represent resource consumption estimation or requirements for the 

corresponding representation with respect to the visual and aural modalities,* 
respectively, as follows: 

F/={Vi(S, K, t A )}i<Nv 

<4*={ai(S, K, t A )} i<Na 

Thus, the above functions indicate the estimated total resources consumed or 
required by the corresponding display or presentation request. 

In one embodiment, it is useful to normalize the resource consumption 
■ estimations so that they will be in a range of [0,1]. In one embodiment, a value of 1 
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denotes "saturation" of the corresponding resource, while a value of 0 indicates that there 
is no noticeable display effect concerning the corresponding resource. 

In one embodiment, a list of all rendering representatives currently on display or 
being presented is maintained to track the amount of resources that are currently being 
consumed and the amount of resources that are still available to accommodate new 
presentation or display requests. In one embodiment; the list of rendering representatives 
is maintained a database referred to as the most-recently updated display database D(t) 
herein. The entries in D(t) are the rendering representatives each of which is denoted as 
follows: 

Repj = (Address, content, tj, S p j, Op > (j > 0 
where: 

o the address contains the positioning information. Typical examples are: 
- Host(TCP(port))^Aiadio(ch'annel),'Telephony(phon^^ " 

o t j corresponds to the time or moment when the corresponding rendering 
representative commenced. Hence, if t = t j then t A = 0. 

o S p j is a pointer to a corresponding frame in the display semantic network S 

o Op is an operator which includes to types of operations: Add, Delete 
Based upon the above description, in order to determine which combination(s) of 
representations and modalities to be used in a given situation, one of the factors or 
criteria that need to be considered is whether there are sufficient available resources to 
accommodate a candidate combination of representation and modality. Therefore, given 
D(t) at a moment t, the task is to determine D(t+1) when the command is 

< Address, content, t j, S PJ , Kj, Op >. The "Delete" operation is straight forward since 
there is no potential conflict with any rendering representative currently on display. The 
more difficult questions is how to deal with the "Add" operation when the command is 

< Address, content, t j, S p; , K j, Add >. 

As described above, for a given application domain, there can be multiple 
representations in which information can be presented to the user. Each representation K 
has a corresponding priority index or preference score that can be preset by the 
application and changed by the user. In the present discussion, a lower priority index 
corresponds to a higher degree of preference. Hence, a representation K having a 
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priority index of 0 is considered most preferable if it can be accommodated given the 
amount of resources available and the amount of resources required to present the 
respective representation. Accordingly, the representation which has the lowest priority 
index (which is most preferable) should be considered first and the representation which 
has the highest priority index (which is least preferable) should be considered last. In 
other words, the system should attempt to use the more preferable representation if it is 
possible to do so. 

Based upon the representation selection strategy discussed above, the system will 
attempt to determine whether the most preferable representation (or the most preferable 
combination of representation and modality) can be accommodated as follows: 
Using: 

D(t') <- D(t) + < Address, content, t, S pj , 0, Add > 



7 pj 

and:let: J .- ; : ! V'- ' - • " 



w<-£vifo,*,;o) 

Ai *.,0) 

If there is no accepted rendering representative produced through the above iteration 
(which means that the most preferable representation cannot be accommodated due to 
insufficient resources), the process continues as follows: 

1 . Beginning with the next frame in the given S, start reiteration until an acceptable 
solution is found (meaning that the current total resource consumption is within the 
range [0,1]). As described above, since the frames corresponding to the multiple 
representations are sorted in an order corresponding to the degree of preference (e.g., 
sorted by priority index), the various representations will be considered in the order 
of preferences (e.g., more preferable representations are considered or examined 
before less preferable representations) 

2. If there is still no solution resulted in step 1 , then determine whether the least 
recently-touched rendering representative in D(t) could be declined. 

3. If step 2 fails, then either: 
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A. Notify the user of the presentation or display overload (insufficient resources 
to accommodate the new presentation or display request); or 

B. Repeat step 2. 

Figure 2 illustrates a block diagram of one embodiment of a system configuration 
200 according to the teachings of the present invention. The system 200, as shown in 
Figure 2, includes one or more applications 210, a presentation or display semantic 
network 220, a decision maker or decision making unit 230, a first database 240, a 
second database 250, and a plurality of output modalities 260. The one or more 
applications 210 can be networked and/or distributed applications. The first database 
240, in one embodiment, is used to store the user preferences with respect to the multiple 
representations in which information from a particular application domain can be 
presented to the user. As described above, the user may indicate his relative preferences 
for the various representations by specifying or changing a priority index (also called 
preference index or preference score) for each representation associated with a given 
application domain. In one embodiment, the first database 240 is also used to stored the 
cognitive relative parameters that are used to quantify the relevant rendering features 
associated with a given representation. For example, considering the calendar example 
described above, some of the parameters associated with a daily representation of a 
calendar may include the size of the display area or window (e.g., large, medium, small, 
etc.), the position of the display area or window (e.g., center, top, bottom, etc.). The 
display semantic network 220 contains one or more linear list of frames each of which 
corresponds to a particular representation in a given application domain and has a finite 
sequence of quadruple slots as described in detail above. The second database 250 (also 
called the most-recently updated display database) is used to store a list of all rendering 
representatives currently on display or being presented at any moment in time. In 
response to a display or presentation request issued by one of the applications, the 
decision unit 230 will attempt to select a most suitable or most preferable combination of 
representation and modality for the presentation request based upon a set of criteria 
including the user representational and/or modal preferences and the amount of resources 
available. In one embodiment, starting with the combination of representation and 
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modality that is considered most preferable (e.g., the representation having the lowest 
priority index), the decision making unit 230 will determine whether available resources 
are sufficient for a particular combination of representation and modality until available 
resources are sufficient to accommodate at least one combination of representation and 
modality or until all combinations have been exhausted. As described above, the 
decision making unit 230 uses the list of rendering representatives maintained in the 
second database 250 to determine whether available resources are sufficient for any 
particular combination of representation and modality, based upon the amount of 
resources currently in use and the amount of resources required for the rendering of the 
particular combination being considered. In one embodiment, as described above, in the 
event that there are not sufficient resources for any combination of representation and 
modality, the decision unit 230 may determine whether the least recently touched 
v^?^t?P^^% m ^jteP!^tC^^ recces. If not, the,decision.unit v .. 
230 may notify the user of the display overload (i.e., available resources are not 
sufficient to accommodate the presentation or display request) or alternatively, may 
continue to check whether the least recently touched rendering representative could be 
declined. 

Figure 3 shows a flow diagram of one embodiment of a method 300 according to 
the teaching of the present invention. At block 3 10, a user in a system is allowed to 
specify a relative preference for each representation in a set of multiple representations in 
which information from a particular application domain can be presented to the user. At 
block 330, the user is also allowed to configure one or more parameters associated with 
each representation to quantify one or more rendering features of the respective - 
representation in one or more modalities in which information from the particular 
application domain can be presented to the user. At block 330, in response to a new 
information presentation request, one or more combinations of representations and 
modalities are selected for the presentation request that can be accommodated by 
available resources, based upon the user preferences and resource consumption 
requirements. 

The invention has been described in conjunction with the preferred embodiment. 
It is evident that numerous alternatives, modifications, variations and uses will be 
apparent to those skilled in the art in light of the foregoing description. 
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CLAIMS 

What is claimed is: 

1 . A method comprising: 

allowing a user in a system to specify a relative preference for each 
representation in a set of representations in which informatipn generated from an 
application domain can be presented to the user; 

allowing the user to configure one or more parameters associated with each 
representation to quantify one or more rendering features corresponding to the respective 
representation in one more modalities in which information from the application domain 
can be presented to the user; and 

in response to a new information presentation request to present information to 
the user, determining one or more combmations of representations aad modalities to be. 
used for the new information presentation request that can be accommodated by 
available resources, based upon a set of criteria including the user preferences and 
resource consumption estimation for the new presentation request. 

2. The method of claim 1 wherein the one or more modalities comprise visual and 
aural modalities. 

3. The method of claim 1 wherein the user relative preferences for the 
representations are maintained in a user preference database. 

4. The method of claim 3 wherein the values for the parameters associated with 
each representation are maintained in the user preference database. 

5. The method of claim 1 wherein determining the one or more combinations 
comprises: 

in response to the new information presentation request, determining which types 
of modalities to be used to accommodate the presentation request, based upon the user 
preferences and the resource consumption estimation for the new presentation request. 
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6. The method of claim 5 wherein determining the one or more combinations 
comprises: 

determining whether available resources are sufficient for a particular 
combination of representation and modality, starting with the representation considered 
most preferable by the user until available resources are sufficient to accommodate at 
least one combination of representation and modality or until all combinations have been 
exhausted. 

7. The method of claim 6 further comprising: 

maintaining a list of rendering representatives, each rendering representative 
corresponding to a previously executed presentation request, the list of rendering 
representatives being used to determine the amount of available resources in each 

'.modality.--...... ». 4 . <■ ,»,.*«.. . ■ „•..:, •. ; . 

8. The method of claim 6 further comprising: 

if available resources are not sufficient for any combination of representation and 
modality, determining whether the least recently touched rendering representative can be 
declined to free up resources for the new presentation request; and 

performing a first action if the least recently touched rendering representative 
cannot be declined. 

9. The method of claim 8 wherein performing the first action comprises: 
notifying the user of a failure to accommodate the new presentation request due 

to insufficient resources if the least recently touched rendering representative cannot be 
declined. 

10. The method of claim 8 wherein performing the first action comprises: 
continuing to check whether the least recently touched rendering representative 

can be declined. 

11. A method comprising: 
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allowing a first user in a system to specify a relative preference for each 
representation of a set of representations in which information associated with a first 
application domain can be presented to the user; 

allowing the first user to specify a relative preference for each modality of a set 
of modalities in which information associated with the first application domain can be 
presented to the user; 

allowing the first user to specify one or more parameters associated with each 
representations to be used for rendering the respective representation in a particular 
modality; and 

in response to a new information presentation request to present information 
associated with the first application domain to the user, selecting one or more 
combinations of representations and modalities to be used for the new information^ 
presentation request that can be accommodated by available resources, based upoaa; set 
of criteria including the user relative preferences for representations and modalities and 
resource consumption estimation for the new presentation request, 

12. The method of claim 1 1 further comprising: 

determining whether available resources are sufficient for a particular 
combination of representation and modality, starting with the combination of 
representation and modality considered most preferable by the user until available, 
presentation resources are sufficient to accommodate at least one combination of 
representation and modality or until all combinations of representations and modalities 
have been exhausted. 

13. The method of claim 12 further comprising: 

maintaining a list of rendering representatives, each rendering representative 
corresponding to a previously executed presentation request, the list of rendering 
representatives being used to determine the amount of available resources for each 
combination of representation and modality. 

. 14. A system comprising: 
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logic ,o allow a user in the system ,„ specif j , re|ative preference fer ^ 
represemation in a « of repress in which information 8enerated from m 
application domain can be presented to the user; 

logic to allow (he user to configure one or more parameters associated with each 
jsentatton to ,ua„«i* one or more rendering features corresponding to the respective 
re „„ ,„ o, more modWities in which infection from the application domain 
can be presented to the user; and 

logic ,o determine, in response t0 , new information ^ 

he application, one or more combinations of representations and modalitiea to be userl 
for the new mfonuafrol, presentalio „ ^ ^ ^ ^ fcy 

resources, baaed upon a M of cataia including to user preferences anJ 

consumption estimation for the new presentation request. 



15. The system of claim 1 4 wherein the one or mote modalities comprise visual and' 
aural modalities. 



16. The system of claim 14 wherein the user relative preferences for the 
representations are maintained in a user preference database. 

17 The system of claim 16 wherein the values for the parameters associated with 
each representation are maintained in the user preference database. 

1 8- The system of claim 14 wherein logic to determine the one or more combinations 
compnses: 

logic to determine whether available resources are sufficient for a particular 
combination of representation and modality, atarting with the representation considered 
most preferable by the user until available resources are sufficient ,o accommodate at 
east one combination of representation and modality or una all representations have 
been exhausted. 
*i 

19. The system of claim 1 8 further comprising: 



19 



WO 02/29603 



PCT/CN00/00301 



logic to maintain a list of rendering representatives, each rendering representative 
corresponding to a previously executed presentation request, the list of rendering 
representatives being used to determine the amount of available resources in each 
modality. 

20. The system of claim 19 further comprising: 

logic to determine whether the least recently touched rendering representative can 
be declined to free up resources for the new presentation request if available resources 
are not sufficient for any combination of representation and modality; and 

logic to perform a first action if the least recently touched rendering 
representative cannot be declined. 

21. .The system of claim 20 wherein logic to perform the first action comprises: 
logic to notify the user of a failure to accommodate the new presentation request 

due to insufficient resources if the least recently touched rendering representative cannot 
be declined. 

22. The system of claim 20 8 wherein logic to perform the first action comprises: 
logic to continue checking whether the least recently touched rendering 

representative can be declined. 

23. A system comprising: 

a first database to store a first user's preferences for a set of representations-each 
of which can be used to present information associated with a first application domain to 
the first user; 

a second database to store a list of rendering representatives each of which 
corresponds to a previously commenced information presentation request; and 

a decision making unit to determine one or more one or more combinations of 
representations and modalities to be used for a new information presentation request 
from the first application domain that can be accommodated by available resources, 
based upon a set of criteria including the user preferences and resource consumption 
estimation for the new presentation request. 
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24. The system of claim 23 wherein the first database is also used to store the first 
user's preferences for a set of modalities in which one or more representations of 
information associated with the first application domain can be presented to the i 



: user. 



25. . The system of claim 24 wherein the first database is also used to store a set of 
parameters associated with each representation, the associated parameters being used to 
quantify one or more rendering features corresponding to the respective representation, 
the parameters being configurable by the first user. 

26. The system of claim 23 wherein the decision making unit comprises: 
logic to determine whether available resources are sufficient for a particular 

combjna^ combination considered *- 

most preferable by the user until available resources are sufficient to accommodate at 
least one combination of representation and modality or until all combinations of 
representations and modalities have been exhausted. 

27. A machine-readable medium comprising instructions which, when executed by a 
machine, cause the machine to perform operations comprising: 

allowing a user in a system to specify a relative preference for each 
representation in a set of representations in which information generated from an 
application domain can be presented to the user; 

allowing the user to configure one or more parameters associated with each- 
representation to quantify one or more rendering features corresponding to the respective 
representation in one more modalities in which information from the application domain 
can be presented to the user; and 

in response to a new information presentation request to present information to 
the user, determining one or more combinations of representations and modalities to be 
used for the new information presentation request that can be accommodated by 
available resources, based upon a set of criteria including the user preferences and 
resource consumption estimation for the new presentation request. 
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28. The machine-readable medium of claim 27 wherein the one or more modalities 
comprise visual modality and aural modality. 

29. The machine-readable medium of claim 27 wherein determining the one or more 
combinations comprises: 

determining whether available resources are sufficient for a particular 
combination of representation and modality, starting with the representation considered 
most preferable by the user until available resources are sufficient to accommodate at 
least one combination of representation and modality or until all combinations of 
representations and modalities have been exhausted, 

30. The machine-readable medium of claim 28 further comprising: 

• - . maintaining ajist'of rendering representatives, 'each rendering representative',, 
corresponding to a previously executed presentation request, the list of rendering 
representatives being used to determine the amount of available resources in each 
modality. 
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